Shifting Attention Using a Temporal Difference Prediction Error and High-Dimensional Input

Research on reinforcement learning has increasingly focused on the role of neuromodulatory systems implicated in associative learning. Formulations of temporal difference (TD) learning have gained a great deal of attention due to the similarity of the TD prediction error and the observed activity of dopamine neurons in the primate midbrain. Recent work has attempted to integrate additional neuro-modulatory systems such as noradrenaline and acetylcholine in a TD framework. Additional work has been done to remedy representational issues arising from TD variants that result in incorrect predictions of dopamine activity, as well as to incorporate the TD error signal in models of categorization. In this paper, an actor–critic model incorporating aspects of TD learning and psychological models of attention is described. The development of the model and the behavior of an autonomous agent in a simulated environment are examined and compared with a variant of TD learning lacking an attentional component. The agent learns to behave adaptively due to the shifting of attention to relevant aspects of a high-dimensional input. In contrast, the TD model exhibits perseverative behavior and comparatively slow learning in the same context. It is suggested that real-time models of attention may provide insight into neuromodulatory systems implicated in attention and representational learning.